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ABSTRACT 


This  study  was  initiated  to  identify  and  illustrate  with  a  visual  model  the  key 
student  learning  competencies  that  most  fundamentally  contribute  to  a  student’s  ability  to 
achieve  desired  learning  outcomes  in  a  synchronous,  dispersed  virtual  learning 
environment.  We  answered  the  following  research  questions:  (1)  which  learning 
competencies  can  best  be  leveraged  to  support  learning  in  a  synchronous  virtual-learning 
environment?  (2)  how  can  relevant  learning  competencies  be  transmitted  to  students 
during  learning  events?  (3)  which,  if  any,  individual-level  cognitive  factors  can  be 
extracted  during  the  pre-training  phase  that  positively  influence  students’  learning 
processes?  and  (4)  which  learning  competencies  can  be  added  to  existing  models  on 
learning  in  virtual  environments? 

By  surveying  students  participating  in  the  U.S.  Navy’s  Virtual  Schoolhouse,  we 
were  able  to  attain  and  analyze  quantitative  data.  We  discovered  that  virtual  world 
efficacy  is  a  meta-competency  composed  of  autonomous  learning,  multiple  level 
operation,  and  collaborative  adaptability.  Our  research  findings  suggest  that  a  student’s 
capability  to  understand  and  learn  in  the  virtual  world  is  a  broad  competency  and  is  a 
significant  predictor  of  his/her  success  in  the  virtual  environment.  Based  on  our  findings, 
we  recommend  a  familiarity  period  to  allow  students  to  develop  these  competencies  prior 
to  their  participation  in  virtual  learning. 
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I.  INTRODUCTION 

A.  PROBLEM  STATEMENT 

As  public  pressure  to  minimize  the  United  States’  national  debt  increases, 
proportional  pressure  is  placed  upon  the  Department  of  Defense  (DOD)  and  its  service 
components  (Army,  Navy,  Marines,  or  Air  Force)  to  reduce  its  discretionary  spending. 
For  the  U.S.  Navy,  specifically,  growing  sequestration  techniques  to  reduce  spending 
clash  with  the  Navy’s  training  requirements.  While  budget  constraints  increase,  the 
Navy’s  need  for  effective  and  efficient  training  continues  to  grow. 

The  dollar  cost  of  the  Navy’s  training  is  significant  and  includes  both  quantitative 
and  qualitative  costs — the  cost  of  manpower  to  develop  the  training  curriculums,  travel 
and  lodging  to  learning  or  training  sites,  and  the  opportunity  costs  of  loss  of  manpower 
and  efficiency  at  commands  with  sailors  away  from  their  assigned  duties  receiving 
training.  Additionally,  traditional  training  at  centralized  Navy  learning  sites  has  the 
potential  to  impact  sailors’  morale,  caused  by  traveling  away  from  assigned  job  duties 
and  family  members.  Traditional  classroom  training  also  tends  to  limit  access  for  students 
to  use  tools  relevant  to  the  system  for  which  they  are  receiving  training.  Self-paced 
practice  on  newly-learned  skills  is  also  typically  unavailable  in  a  Navy  classroom  setting. 
Current  face-to-face  (FTF)  classroom  training  may  not  be  the  most  effective  method  of 
training  to  meet  the  Navy’s  needs,  given  sailors’  unique  morale  challenges,  training 
needs,  and  the  growing  concerns  of  budgetary  cognizance. 

Distributed  virtual  learning  may  provide  an  opportunity  to  reduce  costs  while  also 
improving  training  outcomes.  Many  organizations  have  explored  the  use  of  virtual 
technology,  especially  training  organizations  and  academic  institutions.  A  wealth  of 
research  exists  on  asynchronous  dispersed  learning.  This  research  provides  depth, 
providing  opportunities  to  test  and  replicate  existing  scales,  and  to  use  this  current 
research  as  an  established  foundation.  However,  little  research  has  been  conducted  on 
dispersed  synchronous  virtual  learning.  Studies  recently  conducted  by  Kathryn  Aten  and 
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Marco  DiRenzo  conclude  that  training  in  dispersed  synchronous  virtual  environments  can 
be  successful  (Aten  &  DiRenzo,  2014). 

Distributed  learning  as  a  technique  is  different  than  face-to-face  classroom 
training  and  these  differences  must  be  addressed  both  in  format  of  the  training  and  pre¬ 
training  student  priming.  Students  and  instructors  are  more  familiar  with  face-to-face 
training  as  it  retains  the  power  of  status  quo.  It  relies  on  instructor  direction  it  contains 
familiar  visual  communicative  cues,  and  perceptions  and  attitudes  toward  training  design 
contribute  to  the  overall  learning  outcomes  (Aten  &  DiRenzo,  2014).  Most  courses  are 
designed  for  face-to-face  training;  however,  this  delivery  method  may  not  be  the  most 
effective  or  efficient.  The  majority  of  students  are  familiar  with  daily  online 
communication  and  problem  solving.  Students  may  have  enhanced  access  to  learning 
tools  and  the  opportunity  to  expand  or  enhance  online  communication  and  collaborative 
skills  by  using  virtual  learning  applications.  We  must  understand  these  differences  and 
opportunities  in  order  to  design  training  programs  to  accommodate  them  and  to  fully 
leverage  the  benefits  of  dispersed  synchronous  virtual  learning  as  a  viable  training 
approach. 

B.  BACKGROUND 

1.  Current  Navy  Training 

The  U.S.  Navy’s  administrative,  technical,  and  tactical  training  includes  a 
plethora  of  available  mediums.  A  sailor’s  training  begins  with  introductory  basic  training 
and  continues  throughout  the  length  of  his  or  her  naval  career  with  periodic  training, 
technical  skill  courses  for  specific  systems,  and  tactical  knowledge  development.  Sailors 
are  trained  using  traditional  lecture-style  courses  in  addition  to  three-dimensional  virtual 
simulators,  if  necessary,  for  their  specialty.  These  types  of  training  are  typically 
conducted  in  large  learning  site  facilities,  called  “schoolhouses,”  and  are  based  onshore  in 
fleet  concentration  areas  such  as  Norfolk,  Virginia  and  San  Diego,  California. 

Schoolhouse  training  assignments  can  occur  between  operational  duty 
assignments  or  may  involve  a  temporary  assignment  away  from  existing  duties.  Sailors 
may  be  assigned  to  schoolhouses  for  a  period  of  time  ranging  from  a  few  days  to  several 
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months  while  developing  proficiency  in  their  associated  systems.  Temporary  assigned 
duty  away  from  permanent  duty  stations  incur  travel  and  lodging  costs,  often  to  the 
accrued  point  at  which  satellite  schoolhouses  are  manned  such  as  Naval  Submarine 
Training  Center  Pacific  Detachment  Guam.  Advancing  the  availability  and  enhancing  the 
effectiveness  of  virtual  distance  learning  can  support  the  Navy’s  efforts  to  decrease 
overall  spending  on  training  while  increasing  each  sailor’s  technical  proficiency  to 
maintain  mission  readiness. 

The  Navy’s  current  personnel  training  regimen  includes  a  mix  of  live  and 
simulated  training  approaches  to  complete  training  objectives  for  general  military  training 
(GMT),  technical  and  tactical  development,  and  overall  readiness.  Simulated  training  is 
provided  to  sailors  while  at  sea  to  accomplish  the  periodic  training  requirements  for 
topics  such  as  personal  financial  management,  ethics,  information  assurance,  and 
operational  security  practices.  This  training  has  been  under  deepening  scrutiny  as  its 
ineffectiveness  and  costs  have  been  publicized  by  performance  audit  reports  conducted 
by  the  Government  Accountability  Office  (GAO).  In  recent  years,  the  Navy  has  dedicated 
resources  to  improve  training  effectiveness  and  continue  to  reduce  training  costs. 

Each  naval  community — aviation,  surface,  and  submarine — requires  a  unique 
blend  of  live  and  simulated  training  to  maintain  readiness.  Since  submarines  utilize 
rotating  crews  (two  crews  are  assigned  to  one  submarine),  submarine  sailors  not  currently 
deployed  conduct  training  with  shore -based  simulators  prior  to  deployment  because  live 
training  is  not  possible  while  the  submarine  is  deployed  with  the  alternate  crew  (Pickup  et 
al.,  2012).  The  existing  virtual  training  in  the  submarine  community  has  led  to 
completing  training  objectives  required  for  deployment  readiness  and  has  allowed  the 
community  to  move  forward  in  researching  and  developing  effective  virtual  learning 
approaches  for  other  training  needs,  as  well  (Pickup  et  al.,  2012). 

2.  U.S.  Navy’s  Use  of  Distance  Learning  at  Submarine  Learning  Center 

In  effort  to  reduce  training  costs,  the  Navy’s  Submarine  Learning  Center  (SLC) 
has  developed  and  begun  implementing  a  virtual  schoolhouse  (VSH)  to  provide 
synchronous  distance  learning  for  several  submarine-related  training  courses.  The  SLC, 
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in  cooperation  with  various  industry  and  academic  partners,  developed  the  VSH  in  2013 
to  provide  a  common  training  platform  for  its  distributed  learning  needs  (Aguiar  & 
Wohlgemuth,  2016).  Students  participating  in  the  VSH  environment  are  represented  by 
avatars  and  may  communicate  with  one  other  and  the  instructor  in  real-time  in  addition  to 
interacting  with  the  various  learning  features  and  tools  designed  within  the  VSH,  such  as 
training  aids,  relevant  technical  publications,  and  learning  exhibits.  The  learning 
resources  available  to  students  and  immersive  technology  contribute  to  the  VSH’s  goal  of 
instilling  a  sense  of  presence  in  students  so  they  focus  less  on  their  physical  location  and 
more  on  the  virtual  training  environment. 

C.  PURPOSE 

The  following  items  constitute  the  purpose  of  this  research.  First,  we  will 
determine  which  VSH  features  and  tools  are  most/least  valued  by  Navy  students  so  we 
can  better  understand  which  features  students  find  most  conducive  to  virtual  learning. 
Second,  we  will  investigate  how  instructor  styles  and  tools  contribute  to  students’ 
learning  success  in  the  VSH.  Third,  we  will  identify  key  learning  competencies  that  best 
address  valued  features  so  the  appropriate  features,  coupled  with  the  identified  learning 
competencies,  are  emphasized  during  training.  Fourth,  we  will  build  upon  previously 
developed  models  which  depict  virtual  environment  learning  competencies  and  their 
anticipated  outcomes.  Finally,  we  will  discuss  potential  ways  these  learning  strategies  can 
most  effectively  be  transmitted  to  students  prior  to  their  entering  the  virtual  schoolhouse 
for  actual  training  to  ensure  future  students’  success. 

D.  RESEARCH  QUESTIONS 

This  study  and  the  conclusions  derived  from  the  findings  will  be  directed  by  three 
research  questions.  First,  we  must  answer  the  question,  “Which  learning  competencies 
best  support  learning  in  a  synchronous  virtual-learning  environment?”  To  do  so,  we  will 
test  previously  identified  learning  competencies,  add  any  new  and  relevant  competencies, 
and  test  and  replicate  existing  scales  for  evaluating  the  effectiveness  of  the  VSH. 

Second,  we  will  answer  the  question,  “How  can  relevant  learning  competencies 

be  transmitted  to  students  during  learning  events?”  This  leads  to  directly  to  our  third 
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research  question,  “Which,  if  any,  individual-level  cognitive  factors  can  be  extracted 
during  the  pre-training  phase  that  positively  influence  students’  learning  process?” 

The  fourth  and  final  research  question  we  seek  to  answer  is,  “Which  learning 
competencies  can  be  added  to  existing  models  on  learning  in  virtual  environments?” 

E.  APPROACH 

The  use  of  previous  studies  of  synchronous,  dispersed  learning  will  be  used  for 
background  information  and  study  guidance.  Relevant  studies  we  will  use  include,  but 
will  not  be  limited  to,  organizational  behavior,  business  management,  and  virtual 
education  technology.  Visual  observations  of  the  VSH  during  live  student  engagement 
and  course  offering  will  orient  the  researchers  to  the  VSH  technology.  Students 
participating  in  courses  using  the  VSH  will  be  surveyed,  and  volunteering  students  will 
be  interviewed  to  provide  specific  feedback.  Survey  and  interview  data  will  be  analyzed 
to  determine  the  VSH  features  and  tools  most  valued  by  students.  Finally,  we  will 
develop  an  enhanced  student  learning  competencies  model. 

F.  ORGANIZATION 

This  report  begins  with  this  introduction,  which  provides  background  information, 
the  report’s  purpose  and  approach,  and  the  report’s  research  questions.  Following  this 
introduction  is  a  literature  review  that  examines  the  findings  of  relevant  studies  of  the 
U.S.  Navy’s  requirement  for  virtual  training,  influential  learning  factors  in  virtual 
learning,  observed  competencies  in  synchronous  virtual  collaboration,  and  factors 
affecting  training  in  synchronous  dispersed  virtual  environments.  Next,  the  method 
chapter  will  discuss  the  setting  and  methodology  under  which  we  conducted  our  studies. 
The  following  chapter  will  lists  all  findings  associated  with  our  research.  This  report  will 
end  with  conclusions,  implications,  and  suggestions  for  future,  related  research. 

G.  SUMMARY 

In  response  to  the  Navy’s  challenges  regarding  budgetary  constraints,  the 
Submarine  Learning  Center  has  developed  a  virtual  schoolhouse  to  facilitate  the 
reduction  of  travel  and  lodging  costs  associated  with  required  naval  training  courses. 
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Already  developed  and  implemented,  this  report  will  ascertain  the  virtual  schoolhouse’s 
features  and  tools  from  the  student  perspective.  The  findings  of  this  report  will  include  an 
illustrative  model  of  the  learning  competencies  and  strategies  the  Navy  could  incorporate 
in  its  use  of  the  virtual  schoolhouse  in  order  to  ensure  effective  and  efficient 
synchronous,  dispersed,  and  virtual  learning  while  continuing  to  sequester  costs  incurred 
from  temporarily  assigning  sailors  away  from  their  permanent  duty  stations  for  training. 
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II.  LITERATURE  REVIEW 


A.  LEARNING  FACTORS  IN  SYNCHRONOUS  VIRTUAL 

ENVIRONMENTS 

Students’  personal  perceptions  of  the  virtual  learning  environment  (VLE)  coupled 
with  students’  personality  factors  were  found  to  determine  virtual  training  effectiveness 
based  on  cognitive  processes  (Spears,  2014).  Spears’s  study  included  research  in  the 
disciplines  of  psychology,  education,  and  information  systems  to  identify  three  categories 
which  affect  learning  outcomes  (2014,  p.  32).  The  three  categories  were  student 
perceptions,  student  preparation,  and  mediation  processes  (Spears,  2014,  p.  32).  These 
factors  were  found  to  contribute  to  students’  successfully  achieving  desired  learning 
outcomes.  Using  the  three  broad  categories  stated  above,  Spears  further  refined  the 
factors  which  contribute  to  successful  student  learning  outcomes. 

Student  personality  factors  include  core  self-evaluation,  goal  orientation,  and 
regulatory  focus  (Spears,  2014).  A  student’s  core  self-evaluation  involves  that  student’s 
self-perceptions  regarding  his  or  her  personal  capabilities,  self-esteem,  and  ability  to 
control  his  or  her  own  destiny.  Students  who  are  confident  in  their  own  abilities  are  more 
likely  to  succeed  academically  than  students  who  are  not  confident  in  their  abilities.  This 
likelihood  of  academic  success  also  translates  to  the  VLE.  Goal  orientation  is  measured 
by  a  student’s  motivation  to  achieve  high  performance  scores  or  receive  the  most 
knowledge  from  the  learning  experience.  A  student’s  regulatory  focus  is  essentially  the 
link  between  motivation  and  goal  achievement  (Spears,  2014,  p.  xvii). 

Comprehending  a  student’s  perceptions  about  a  VLE  provides  insight  to 
understanding  student  performance  during  learning  events.  The  perspective  of  the 
students  measures  the  effectiveness  of  a  specific  VLE  (Spears,  2014).  Perceived  system 
performance  addresses  whether  students  find  a  VLE  system  to  be  helpful  and  convenient, 
utilizing  factors  such  as  usefulness,  ease  of  use,  technical  performance,  and 
communication  richness  (Spears,  2014). 
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Students’  perceptions  of  their  instructor  in  a  VLE  create  a  perception  of  the 
quality  of  execution  with  factors  such  as  instructor  attitudes,  interactivity  via  the  VLE, 
and  technological  competence  (Spears,  2014).  Instructors’  attitudes  toward  VLE 
technology  affect  student  attitudes  towards  VLE.  Instructors  use  of  interactivity  features 
to  complement  their  teaching  styles  while  interacting  with  students  via  the  VLE  is  crucial 
to  well-received  student  perceptions.  Additionally,  an  instructor’s  technical  ability  to 
effectively  operate  the  VLE  technology  is  also  imperative  to  positive  student  perceptions. 

Mediating  processes  include  cognitive  attitude  and  cognitive  engagement. 
Attitude,  by  definition,  is  a  person’s  feelings,  whether  negative  or  positive,  regarding 
performing  a  specific  task  or  behavior  (Spears,  2014,  p.  18).  Cognitive  attitude  mediates 
the  influence  of  a  student’s  perceived  usefulness  and  perceived  ease  of  use  of  VLE 
technology  (Spears,  2014,  p.  18).  The  definition  of  cognitive  engagement  is  “the  quality 
of  students’  psychological  engagement  in  academic  tasks,  including  their  interest, 
ownership,  and  strategies  for  learning”  (Spears,  2014,  p.  17).  Cognitive  engagement  is  a 
predictor  of  a  student’s  academic  success  as  it  depends  on  the  quality  of  a  student’s 
efforts  to  understand  or  master  a  learning  outcome. 

Personality  factors  and  perceptions  influence  student  cognitive  processes,  which 
enable  students  to  achieve  learning  outcomes,  as  shown  in  Figure  1.  Spears’s  study 
concluded  that  a  synchronous,  dispersed  learning  environment  with  restrictive  learner 
control  was  the  most  plausible  Navy  training  application. 
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Figure  1.  Conceptual  Framework  of  Learning  in  Synchronous  Virtual 
Environments.  Source:  Spears  (2014). 


B.  COGNITIVE  LEARNING  PROCESSES  AND  OUTCOMES  IN  THE 

VIRTUAL  SCHOOLHOUSE 

This  study  built  directly  on  Spears’s  model  and  conclusions  and  tested  some  of 
the  conceptual  relationships  outlined  by  Spears  by  exploring  the  following  two  questions: 

1.  “What  role  do  cognitive  factors  and  processes  play  in  performance  and 
satisfaction  outcomes  of  education  and  training  conducted  in  virtual 
worlds?”  and; 

2.  “What  adaptations  to  behavior  and  communication  can  improve  education 
and  training  in  virtual  worlds?”  (Aten  &  DiRenzo,  2014,  p.  2). 

Aten  and  DiRenzo’ s  study  at  the  Fleet  Anti-Submarine  Warfare  Training  Center 
compared  the  virtual  schoolhouse  to  Face-to-Face  (FTF)  training  and  were  able  to 
synthesize  Spears’s  conceptual  model  into  three  core  competencies.  Students’  success  in 
achieving  learning  outcomes  in  VLEs  were  found  to  be  dependent  upon  student  cognitive 
competencies,  perceptions  of  the  learning  system  and  instructors,  and  cognitive  processes 
(Aten  &  DiRenzo,  2014). 

A  student’s  cognitive  competencies  consist  of  cognitive  capacity,  multimedia  self- 
efficacy,  self-regulated  learning,  and  regulatory  focus  (Aten  &  DiRenzo,  2014).  Aten  and 
DiRenzo’s  study  measured  students’  intellectual  capability,  or  cognitive  capacity,  by 
capturing  test  performance  scores  on  previous  modules  (2014,  p.  4).  Comfort  and 
competency  operating  communications  technologies  identifies  a  student’s  multimedia 
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self-efficacy.  Self-regulated  learning  is  a  student’s  ability  to  direct  autonomous  and 
proactive  learning.  As  found  in  Spears’s  research,  regulatory  focus  is  a  student’s 
motivation  by  either  promotion  or  prevention.  Promotion  focus  encompasses  a  student’s 
motivation  to  earn  awards  and  achieve  goals,  whereas  prevention  focus  is  a  student’s 
motivation  to  avoid  punishment  or  failure  (Aten  &  DiRenzo,  2014,  p.  4). 

Perceptions  of  the  learning  system  includes  students’  perceptions  of  the 
usefulness  and  ease  of  use  of  the  VSH,  as  well  as  features  which  enable  interactivity  with 
the  instructor  and  learning  materials  access.  Students’  perceptions  of  the  instructor’s 
quality  and  knowledge,  plus  the  students’  perceptions  of  the  instructor’s  attitude  toward 
interacting  within  the  VSH,  all  influence  students’  overall  perceptions  of  the  VSH. 

Similar  to,  and  deriving  from,  Spears’s  findings,  Aten  and  DiRenzo  state 
cognitive  engagement  and  cognitive  attitudes  comprise  the  cognitive  processes  which 
identify  student  interest  and  beneficiary  attitudes  towards  learning  in  the  VSH. 

Utilizing  Spears’s  conceptual  framework,  Aten  and  DiRenzo  expanded  and  tested 
individual-level  factors  which  contribute  to  learning  and  satisfaction  success  in  virtual 
learning  environments.  Although  the  sample  size  was  small,  study  findings  suggest  that 
“virtual  environments  are  a  viable  alternative  for  the  delivery  of  education  and  training” 
(Aten  &  DiRenzo,  2014,  p.  8).  Performance  scores  on  knowledge  and  practical  tests  were 
slightly  higher  for  VSH  students,  while  FTF  students  reported  more  positive  attitudes  in 
classroom  and  lab  training.  Additionally,  Initial  Entry  sailors  did  not  perform  as  well  as 
Fleet  Returnees;  and  cohorts  with  a  mix  of  Initial  Entry  sailors  and  Fleet  Returnees 
scored  higher  than  the  cohort  of  all  Initial  Entry  sailors.  Using  these  findings,  plus  those 
from  Spears’s  study,  the  Aten-DiRenzo  model,  depicted  in  Figure  2,  was  created  to 
illustrate  the  causation  of  the  predictive  relationships  between  the  four  student  cognitive 
factors,  student  perceptions  of  the  learning  system  and  instructors  within  it,  cognitive 
processes,  and  student  learning  outcomes. 

The  study  conducted  by  Aten  and  DiRenzo  recommends  student  preparation  for 
learning  in  VSH  with  pre-training  and  priming  (2014,  p.  19).  Due  to  the  self-regulatory 
requirements  of  virtual  learning,  students  should  be  trained  with  basic  skills  to  best  take 
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control  of  their  learning  and  plan,  monitor,  and  evaluate  the  effectiveness  of  their 
learning  throughout  the  VSH  experience.  Instructor  preparation  should  include  priming 
them  to  express  enthusiasm  regarding  teaching  in  the  VSH  and  be  trained  competently  on 
the  operation  of  the  VSH  features  and  tools  (Aten  &  DiRenzo,  2014,  p.  20).  Lastly,  the 
study  recommends  learning  system  design  should  be  enhanced  to  increase  student- 
instructor  interaction  between  one  another  and  the  learning  content  (Aten  &  DiRenzo, 
2014,  p.  20). 


Student  Perceptions 


Student  Cognitive  Factors 
^^CognitiveCapacity^^^J 

|^^Multi-Medi^Self-Efficacy^J 

Self-Regulated  Learning 
|^^^^RegulatoiyFocus^^^J 


Learning  System 

•  Perceived  Ease  of  use 

•  Perceived  Usefulness 

•  Interactivity  with  the 
Instructor 

•  Interactivity  with  learning 
content 


Instructor 

•  Instructor  quality 

•  Instructor  attitude  toward 
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Cognitive  Processes 


COGNITIVE 

ENGAGEMENT 


ATTITUDE 


Student 

Learning  Outcomes 


PERFORMANCE 

•  Classroom 

•  Lab  Practical 


SATISFACTION 


Figure  2.  Model  of  Learning  in  Virtual  Environments. 

Source:  Aten  and  DiRenzo  (2014). 


C.  COMPETENCIES  FOR  WORKING  IN  VIRTUAL  WORLDS 

The  next  step  in  the  in  the  evolution  of  this  research  stream  was  the  study 
conducted  by  Aten,  DiRenzo,  Stanko,  Lahneman,  Nardon,  and  Richter  in  2014.  This 
study  scrutinized  two  additional  studies  and  developed  the  framework  of  four 
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competencies  attributable  to  virtual  learning  effectiveness.  The  Aten  et  al.  study  drew  on 
three  exploratory,  qualitative  studies,  which  examined  teams  conducting  work  in  virtual 
worlds  to  identify  key  competencies  for  effective  virtual  work. 

The  first  of  the  three  studies,  “Working  Out  Loud,”  was  based  on  an  in-depth 
study  of  a  case  of  successful  collaborative  knowledge  work  by  a  distributed  group 
working  in  a  virtual  world  (Aten  et  al.,  2014).  “Plastic  Prometheus”  was  the  second  study 
and  analyzed  in-depth  interviews  of  individuals  working  in  three-dimensional  virtual 
environments  to  examine  affordances  that  emerge  around  daily  use  of  technology  for 
work  and  how  they  relate  to  work  routines  (Aten  et  al.,  2014).  The  third  study,  and  the 
one  on  which  our  research  is  built,  was  conducted  during  a  test  of  the  Navy’s  Submarine 
Learning  Center  (SLC)  virtual  schoolhouse  (VSH).  According  to  the  study,  “the  VSH 
enabled  distributed  instructors  and  students,  represented  by  avatars,  to  interact  with  each 
other  and  to  access  live  tactical  systems  across  distance  in  a  3D  virtual  environment” 
(Aten  et  al.,  2014). 

Findings  from  these  three  studies  suggest  four  learning  competencies  which 
involve  key  knowledge,  skills,  and  attitudes  that  are  required  for  effective  learning  in 
virtual  environments.  These  four  competencies  identified  were  a  student’s  ability  to  “(1) 
articulate  thoughts  and  actions  in  ways  that  differ  from  face-to-face  communication,  (2) 
use  technology  to  conduct  communication  on  multiple  (often  simultaneous)  levels  and  in 
multiple  timeframes,  (3)  accept  contextual  ambiguity  and  adopt  alternate  perspectives 
and  identities,  and  (4)  develop  and  maintain  positive  affective  and  cognitive  attitudes 
toward  virtual  world  technology”  (Aten  et  al.,  2014,  p.  25). 

Since  communicating  in  virtual  environments  lacks  the  physical  cues  and  body 
language  of  face-to-face  communication,  thoughts  and  actions  must  be  articulated 
differently  in  order  to  be  effective.  Individuals  working  in  virtual  worlds  need  to  be  able 
and  willing  to  adapt  their  behavior  for  working  in  virtual  environments.  Individuals  need 
to  be  able  to  articulate  their  thoughts  and  actions  as  they  conduct  work  so  that  other  team 
members  can  construct  a  rich  visual  representation  of  the  action  being  taken.  They  need 
to  make  evident  their  strategy  and  thought  processes  behind  the  action  in  the  absence  of 

cues  that  are  often  present  in  a  face-to-face  setting. 
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The  use  of  technology  to  conduct  communication  on  multiple  and/or  simultaneous 
levels  also  creates  unique  challenges  for  individuals  interacting  in  virtual  environments. 
Those  conducting  work  in  virtual  environments  require  skills  to  communicate  layers  of 
meaning  and  maintain  the  flow  of  work  and  interaction  despite  barriers  as  well  as  the 
absence  of  cues  and  the  routines  they  trigger.  Individuals  must  be  able  to  navigate  across 
multiple  technological  platforms  which  support  and  sustain  interaction. 

Individuals  must  be  able  to  model  and  visualize  new  products,  or  to  be  able  to 
walk  into  life-size  prototypes  and  understand  how  different  elements  are  spatially  related 
to  one  another.  This  allows  individuals  to  readily  accept  contextual  ambiguity  and 
embrace  different  perspectives  and  identities. 

Without  the  ability  to  maintain  positive  affective  and  cognitive  attitudes  towards 
the  technology  that  is  used,  individuals  participating  in  the  VSH  are  less  open  and  unable 
to  adapt  their  behaviors  in  the  ways  that  are  critical  for  successful  virtual  learning. 
Therefore,  it  is  imperative  that  individuals  develop  and  maintain  positive  attitudes  toward 
virtual  environment  learning  technologies. 

In  sum,  the  study  conducted  by  Aten  et  al.  derived  four  competencies  from  three 
previous,  relevant  studies.  One  singular  competency  is  insufficient  alone  for  success  in 
the  VSH.  Each  competency  is  intertwined  and  must  be  present  for  success  in  the  VSH.  In 
addition  to  the  studies  discussed  above,  studies  by  Nakayama,  Yamamoto  &  Santiago, 
“The  Impact  of  Learner  Characteristics  on  Learning  Performance  in  Hybrid  Courses 
among  Japanese  Students,”  and  M.  Bower,  “Synchronous  Collaboration  Competencies  in 
Web-Conferencing  Environments,”  also  offered  additional  insight  in  to  potential 
competencies  leading  to  student  success  in  the  VSH. 

D.  THE  IMPACT  OF  LEARNER  CHARACTERISTICS  ON  LEARNING 

PERFORMANCE  IN  HYBRID  COURSES 

In  this  2007  study,  Nakayama,  Yamamoto,  and  Santiago  examined  the  learning 
characteristics  and  performance  of  84  students  at  Tokyo’s  Institute  of  Technology  by 
analyzing  students’  test  modules  performance  scores  (p.  195).  Students  self-assessed  their 
online  learning  experience  to  identify  possible  causal  paths  to  learning  performance 
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based  on  learning  characteristics.  Students  were  Bachelor’s  program  freshmen  and 
Masters’  students  in  a  hybrid  course  that  mixed  face-to-face  learning  with  online 
learning. 

Based  on  the  data  Nakayama  et  al.  gathered,  three  learning  factors  were 
constructed:  e-Learning  overall  evaluation,  learning  habits,  and  learning  strategies 
(Nakayama  et  al.,  2007,  p.  199).  These  three  learning  factors  which  can  be  aligned  with 
and  support  several  items  in  the  Aten-DiRenzo  (2014)  Model  of  Learning  in  Virtual 
Environments.  Factor  1,  “overall  evaluation  of  e-learning  experience,”  includes  items  that 
measure  perceived  ease  of  use  (‘E-Learning  is  easy  to  follow  and  understand’)  and 
perceived  usefulness  (“Online  materials  are  useful  to  me”)  (Nakayama  et  al.,  2007,  p. 
199).  Factor  2,  “learning  habits,”  includes  items  which  measure  self-regulated  learning 
(“I'm  a  conscientious  student,”  and  “It  is  my  habit  to  do  learning  preparation  and 
review”)  (Nakayama  et  al.,  2007,  p.  199).  Factor  3,  “learning  strategies,”  includes  items 
that  also  measure  self-regulated  learning  (“I  have  my  own  method  and  way  of  learning” 
and  “I  have  my  own  strategies  on  how  to  pass  a  course”)  (Nakayama  et  al.,  2007,  p.  199). 

Differences  were  discovered  in  conscientiousness  between  students  who  earned 
final  grades  of  “A”  and  “B”  in  the  course,  suggesting  student  personality  affects  final 
grades  (Nakayama  et  al.,  2007,  p.  205).  Master’s  students  had  higher  “learning  strategy” 
scores  than  Bachelor’s  students,  indicating  that  Master’s  students  may  have  positively 
recognized  the  benefits  of  online  courses  (Nakayama  et  al.,  2007,  p.  205).  These  students 
likely  developed  strategies  (e.g.,  “access  to  online  modules  during  that  time  of  day  when 
they  are  most  ready  to  study  and  learn”)  themselves  as  an  application  of  self-regulatory 
focus  (Nakayama  et  al.,  2007).  Learner  characteristics  were  found  to  have  affected 
learning  experience  and  performance.  Learner  characteristics — “motivation,  personality, 
thinking  styles” — and  learners’  perception  of  their  e-Learning  experiences  were 
measured  at  the  beginning  and  end  of  the  term,  and  master  students’  perception  of  their  e- 
Learning  experience  increased  significantly  throughout  the  course  (Nakayama  et  al., 
2007,  p.  205). 
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E.  SYNCHRONOUS  COLLABORATION  COMPETENCIES  IN  WEB¬ 
CONFERENCING  LEARNING  ENVIRONMENTS 

M.  Bower's  study  conducted  in  2010  observed  four  levels  of  online  collaborative 
competencies  with  impact  the  online  learning  process:  operational,  interactional, 
managerial,  and  design  (M.  Bower,  2010,  p.  63).  The  study  analyzed  26  college  students 
over  three  academic  semesters  using  multi-modal  pedagogies  to  refine  the  discovered 
competencies  (M.  Bower,  2010,  p.  67). 

The  first  competency  identified,  operational,  is  one’s  ability  to  operate  the 
functions  of  the  collaborative  environment  technology  (M.  Bower,  2010,  p.  76).  These 
are  the  easiest  to  develop  due  to  the  relatively  simple  repetition  of  tasks  in  the  virtual 
environment;  however,  operational  competencies  require  additional  practice  to  become 
proficient.  Such  repetition  allows  for  quick  adaptation  for  both  students  and  instructors. 
Common  operational  functions,  such  as  text  chat,  whiteboard  use,  screen  board  rotation, 
and  the  uploading  and  downloading  of  files,  are  executed  by  the  student  and  instructor 
early  and  often  enough  they  can  facilitate  an  organically  developed  adaptation  of  many 
operational  competencies. 

Interactional,  the  second  competency,  is  one’s  “ability  to  effectively  interact  to 
perform’’  tasks  and  solve  problems  using  the  collaborative  technology  by  forming  an 
understating  of  who  is  manipulating  the  virtual  environment  and  for  what  purpose  (M. 
Bower,  2010,  p.  76).  Understanding  who  is  contributing  to  the  virtual  learning 
environment  helps  to  “overcome  distributed  process  loss  by  providing  people  with”  an 
awareness  of  others’  actions  and  the  center  of  concentration  (M.  Bower,  2010,  p.  76).  For 
example,  the  student  will  react  differently  if  the  instructor  is  manipulating  an  element  of 
the  VSH  as  opposed  to  a  fellow  student.  Capturing  this  information  can  be  accomplished 
by  adding  initials  to  notepad  contributions  and  using  audio  to  indicate  the  focus  of 
attention. 

The  third  competency,  managerial,  is  the  ability  to  manage  the  class  including 
effective  technology  use  and  interaction  (M.  Bower,  2010,  p.  76).  This  competency 
applies  to  instructors  and,  if  poorly  applied,  could  cause  negative  ripple  effects 
throughout  the  classroom. 
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The  fourth  and  final  competency  identified  by  M.  Bower  is  design,  defined  as  the 
ability  to  select  and  organize  tools  to  optimize  interaction  and  support  activity 
management  (M.  Bower,  2010,  p.  76).  Understanding  the  representational  and 
interactional  potential  of  the  different  tools  provided  in  the  virtual  classroom  allows  for 
the  maximum  participation  and  interaction  between  the  instructor  and  students.  Used  not 
only  to  assist  in  fostering  the  developed  of  the  Interactive  competency,  but  in  all 
functional  aspects  of  the  VSH.  The  arrangement  of  different  tools  can  be  conducted  by 
either  the  instructor  prior  to  the  beginning  of  class  or  by  the  students  during  a  learning 
episode.  Since  different  tools  are  required  for  different  classes,  effective  design  of  the 
VSH  requires  that  both  the  instructor  and  the  students  understand  the  different  functional, 
representational  and  interactional  potential  of  tools. 

Competencies  derived  from  synchronous  collaborative  learning  are  dependent 
upon  the  type  of  pedagogies  being  applied  (M.  Bower,  2010,  p.  80).  Instructor-centered 
learning  requires  lower  levels  of  managerial  competencies  of  the  instructor,  but  higher 
operational  competencies  of  students.  Student-led  learning  requires  the  instructor  to  have 
greater  managerial  and  design  competency,  while  students  will  likely  exercise  some 
managerial  competencies  as  well. 

F.  SUMMARY  OF  LITERATURE  REVIEW 

In  assessing  and  comparing  the  relevant  data,  we  discovered  seven  learning 
competencies  which  may  best  predict  student  learning  outcomes  in  VLEs.  All  four  of 
Aten’s  competencies,  Aten  and  DiRenzo’s  self-regulated  learning,  and  M.  Bower’s 
operational  and  interactional  competencies  would  likely  best  predict  student  learning 
outcomes  (Aten,  2014;  Aten  &  DiRenzo,  2014;  M.  Bower,  2010).  In  addition,  Aten’s 
four  competencies — articulating  thoughts  and  actions  in  a  collaborative  environment, 
using  the  technology  to  communicate  on  multiple  levels,  accepting  contextual  ambiguity, 
and  maintaining  positive  affective  and  cognitive  attitudes — signal  prediction  of  students 
freely  accepting  training  and  collaborating  via  the  VLE  technology  (2014).  Their  physical 
and  cognitive  ability  to  collaborate  can  also  be  predicted  using  M.  Bower’s  operational 
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and  interactional  competencies.  Finally,  a  student’s  self-regulated  learning  abilities  are 
required  to  successfully  achieve  desired  VLE  learning  outcomes. 

G.  THEORETICAL  MODEL  DEVELOPMENT 

Based  on  the  relevant  literature  reviewed,  we  expected  to  see  strong  correlations 
among  all  seven  identified  learning  competencies  listed  above.  Due  to  the  specific  nature 
of  Aten’s  four  competencies,  we  predicted  they  would  likely  be  more  descriptive  of  the 
two  competencies  identified  by  M.  Bower.  Additionally,  the  competency,  self-regulatory 
learning,  was  expected  to  have  relatively  strong  correlations  to  the  other  competencies 
measured  in  the  study. 

Figure  3  illustrates  our  anticipated  adapted  model,  using  Spears  and  Aten’s 
models  from  Figures  1  and  2,  respectively,  and  refining  the  learning  competencies  to 
demonstrate  the  influence  of  the  seven  anticipated  competencies  we  have  identified.  All 
seven  competencies  are  listed  under  “Student  Competencies”  and  can  likely  be  taught 
and/or  improved  upon  before  a  student  begins  VSH  course  work.  “Student  Perceptions” 
include  the  student’s  perceptions  of  the  VSH  learning  system  and  the  instructor,  which 
cannot  be  taught  or  primed  prior  to  engaging  in  VSH  courses.  Student  competencies  and 
perceptions  will  influence  their  cognitive  processes — cognitive  engagement  and 
attitude — ultimately  affect  the  desired  learning  outcomes,  both  performance  and 
satisfaction. 
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Figure  3.  Theorized  Model  of  VSH  Learning  Competencies 
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III.  METHOD 


A.  PROJECT  PURPOSE 

The  overall  purpose  of  this  project  is  to  create  an  illustrative  model  of  the  learning 
competencies  which  contribute  to  successful  learning  in  the  VSH.  Using  the  Aten  and 
DiRenzo  study,  “Assessing  the  potential  of  virtual  worlds  for  Navy  Training  and 
Education:  cognitive  learning  processes  and  outcomes  in  the  virtual  schoolhouse”  as  a 
basis  for  this  project,  we  expanded  the  sample  size  of  students  surveyed  from  the  original 
36  students  to  39  additional  students.  We  replicated  Aten  and  DiRenzo’s  study  by 
observing  three  training  sessions  and  administering  surveys  to  students.  While  refining 
the  Aten-DiRenzo  model,  we  will  analyze  our  research  findings  to  add  new,  relevant 
learning  competencies. 

B.  SETTING 

The  students  were  set  in  classrooms  which  contained  desktop  computer  systems 
that  were  connected  to  the  VSH  program  network.  The  classrooms  were  located  in 
Groton,  Connecticut;  Kings  Bay,  Georgia;  and  Norfolk,  Virginia.  Classroom  organization 
varied  between  each  location  and  training  session — some  sessions  had  a  senior  enlisted 
proctor  present  to  address  questions  regarding  operating  the  VSH  and  computers,  some 
sessions  did  not  have  any  staff  members  present  for  the  training.  Students  in  the  three 
geographical  locations  mentioned  above  participated  in  three  different  courses,  each  three 
days  in  length,  and  conducted  solely  via  the  VSH.  From  the  students  participating  in  the 
VSH,  volunteers  were  identified  to  remotely  provide  via  interviews  at  a  later  date  their 
personal  observations  and  perceptions  while  using  the  VSH. 

Site  visits  were  conducted  in  May  to  observe  students’  participation  in  the  VSH 
and  administer  surveys.  Surveys  administered  in  May  were  conducted  by  hand.  A  total 
of  39  students  participated  in  VSH  courses.  The  VSH  courses  were  hosted  by  an 
instructor  in  Groton,  Connecticut,  and  taught  students  located  in  Groton,  Connecticut; 
Norfolk,  Virginia;  and  Kings  Bay,  Georgia  via  the  VSH  collaborate  computer  system. 
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C.  SURVEY 

The  surveys  were  designed  to  assess  students’  perceptions  of  the  VSH  and 
determine  which  learning  competencies  are  being  transmitted  during  training.  While 
developing  questions  for  the  survey,  we  used  Aten  and  DiRenzo’s  previous  study’s 
survey  as  a  base.  From  that  base,  we  added  questions  which  were  relevant  to  the  virtual 
environment  learning  competencies  previously  identified  during  our  literature  review. 
These  included  the  following  competencies: 

1.  Articulate  thoughts  and  actions  in  a  collaborative  environment  (Aten  et  al., 
2014); 

2.  Use  the  technology  to  communicate  on  multiple  levels  (Aten  et  al.,  2014); 

3.  Accept  contextual  ambiguity  (Aten  et  al.,  2014); 

4.  Develop  and  maintain  positive  affective  and  cognitive  attitudes  (Aten,  et 
al,  2014); 

5.  Operational  (M.  Bower,  2010); 

6.  Interactional  (M.  Bower,  2010);  and 

7.  Self-regulated  learning  (Aten  et  al.,  2014). 

Surveys  were  completed  by  students  before  beginning  their  VSH  orientation  at 
the  very  start  of  the  training  session  and  then  administered  post-training.  The  pre-  and 
post-training  surveys  given  to  students  can  be  found  in  Appendix  A  and  Appendix  B, 
respectively. 

D.  FOLUOW-UP  INTERVIEWS 

At  the  conclusion  of  the  site  visit,  students  at  all  three  locations  were  asked  if  they 
would  be  willing  to  sign  a  waiver,  and  agree  to  participate  in  an  oral  interview  at  a  later 
date.  Eleven  of  39  students  signed  the  waiver;  out  of  the  11  students  who  signed  the 
waiver,  the  researchers  were  able  to  contact  and  interview  five  students.  The  five 
interviews  that  were  conducted  generated  134  minutes  of  narrative  consisting  of  answers 
to  18  pre-determined  questions,  listed  in  Appendix  C.  The  interview  questions  were 
chosen  based  on  their  ability  to  elicit  responses  from  students  that  could  be  associated 
with  the  learning  competencies  identified  in  previous  studies.  Students  were  encouraged 
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to  provide  specific  examples  of  events  that  occurred  during  the  actual  course.  The 
interviews  were  conducted  over  the  phone,  recorded,  and  transcribed. 
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IV.  FINDINGS 


A.  LEARNING  COMPETENCY  INTER-RELATIONSHIPS 

The  data  were  analyzed  using  the  correlational  analysis  among  the  competencies. 
The  competencies  included  the  seven  identified  in  Chapter  II  of  this  study,  as  well  as 
other  competencies  and  competency  subscales  from  Aten  and  DiRenzo’s  previous  study 
designs  (2014).  Appendix  D  illustrates  the  correlation  relationships  between  all 
competencies  studied.  Our  primary  goal  for  this  study  was  to  identify  and  examine  how 
the  student  learning  competencies  inter-relate  with  one  another.  By  examining  the 
correlation  matrix,  we  were  able  to  determine  which  competency  constructs  strongly  and 
positively  influence  one  another.  Table  1  illustrates  the  most  significant  correlations 
between  the  learning  competencies. 


Table  1.  Significant  Correlation  Data  between  Learning  Competencies 
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Correlation  data  with  significance  of  p  <  0.05 


The  correlation  analysis,  coupled  with  the  existing  research  discussed  previously, 
suggest  an  important  key  finding.  Virtual  world  efficacy  (VWE)  appears  to  be  the  most 
overarching  of  all  the  competencies  measured.  As  shown  in  Table  1,  VWE  was  strongly 
correlated  with  internet,  operational,  interactional,  and  ambiguity  competencies,  while 
also  exhibiting  positive  correlations  with  all  other  competencies  studied  (see  Appendix 
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D).  Based  on  this  key  finding,  we  have  re-conceptualized  seven  competencies  identified 
in  Chapter  II.  We  elaborate  on  this  re-conceptualization  below. 

B.  MODEL  RE-CONCEPTUALIZATION 

The  technological  nature  of  the  VSH  suggests  that  the  learning  competencies 
which  may  best  be  developed  in  students  in  order  to  achieve  learning  outcomes  are 
subsets  to  the  students’  ability  to  operate  the  VSH  technology  and  their  comfort  in  doing 
so.  The  importance  of  students’  virtual  world  efficacy  in  their  VSH  success  suggests  that 
it  represents  a  meta-competency,  being  comprised  of  various  other  competencies  that, 
together,  may  enhance  student  learning  in  a  virtual  world.  Hence,  as  shown  in  Figure  4, 
we  present  the  Bower-Richmond  model  of  student  competencies  in  the  virtual  learning 
environment,  which  suggests  that  success  in  the  virtual  learning  environment  is  driven  by 
an  over-arching  competency  (virtual  world  efficacy),  comprised  of  three  distinct,  yet 
related,  dimensions  (autonomous  learning,  multiple  level  operation,  and  collaborative 
adaptability).  We  discuss  these  competencies  in  detail  in  the  following  sections. 
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Figure  5.  Bower-Richmond  Model  of  Student  Competencies  in  the 
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Virtual  world  efficacy  (VWE)  represents  a  student’s  ability  to  understand  and 
learn  in  a  virtual  environment.  Students  must  be  competent  in  using  communications 
technology  in  order  to  successfully  achieve  desired  learning  outcomes  in  the  virtual 
environment.  This  entails  not  just  operating  the  virtual  environment  technology — such  as 
moving  an  avatar,  adjusting  the  zoom  level  of  the  virtual  classroom,  or  navigating  around 
the  virtual  classroom — but  also  maintaining  several  conversations  with  multiple 
participants  in  a  synchronous  chat  and  simple  internet  operations  like  opening  a  web 
browser  or  switching  between  different  windows.  Because  it  encompasses  a  broad  array 
of  capabilities  and  related  to  all  other  competencies,  virtual  world  efficacy  represents  the 
top-tier  learning  competency  based  on  our  findings.  The  second-order  tier  contains 
autonomous  learning,  multiple  level  operation,  and  collaborative  adaptability. 

Autonomous  learning  represents  a  student’s  ability  to  direct  his  or  her  own 
learning  both  independently  and  proactively.  This  competency  includes  factors  such  as 
the  student’s  capacity  for  self-regulated  learning,  which  has  been  discussed  previously. 
Students  must  be  able  to  make  conscious  and  informed  decisions  without  coercion  or 
prompting  from  instructors  to  learn  effectively  in  the  virtual  environment.  This  is 
especially  true  in  learning  in  the  virtual  world,  when  cues  from  instructors  typical  in  FTF 
learning  may  be  absent. 

Multiple  level  operation  represents  a  student’s  ability  to,  and  confidence 
towards,  using  the  features  and  tools  of  the  virtual  environment.  This  competency  is  an 
expansion  of  Bower’s  (YEAR)  “operational”  competency.  It  includes  simultaneously 
communicating  across  various  virtual  platforms  and  across  multiple  levels ,  such  as  the 
virtual  classroom  and  virtual  practical  laboratory.  Multiple  level  operation  also  requires 
the  student  to  readily  accept  ambiguity  in  the  virtual  world  context  and  embrace  different 
perspectives  and  identities  while  performing  virtual  learning  tasks.  The  subsets  of  this 
competency,  shown  in  Figure  4,  are  primarily  derived  from  Aten  and  DiRenzo’s  (2014) 
study  and  incorporate  the  definitions  of  their  identified  ambiguity  and  multiple  levels 
competencies. 
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Finally,  collaborative  adaptability  represents  as  a  student’s  ability  to  collaborate 
with  the  technology,  instructor,  and  other  students  to  perform  learning  tasks  and  solve 
virtual  problems.  Again,  this  competency  is  an  expansion  and  re-conceptualization  of  M. 
Bower’s  (2010)  interactional  competency  and  is  also  likely  related  to  one’s  multiple  level 
operation  competency  as  well.  Students  must  understand  the  need  for  adaptability  in 
order  to  successfully  interact  with  not  only  the  instructor,  but  their  fellow  students  and 
the  virtual  technology  as  well.  This  requires  adapting  to  ways  of  articulation  so  as  to 
appropriately  convey  and  receive  intended  communications  and  messages.  As  such, 
mastering  communication  via  synchronous  chat  is  vital  to  effective  collaboration. 

C.  COMPETENCY  EFFECTS  ON  COGNITIVE  PROCESSES  AND 

LEARNING  OUTCOMES 

Our  next  goal  was  to  examine  the  relationships  between  these  competencies  and 
both  cognitive  processes  and  final  learning  outcomes  in  the  virtual  world  learning 
environment. 

1.  Cognitive  Processes 

The  correlation  data  suggests  many  relationships  that  influence  a  student’s 
cognitive  processes.  These  processes  include  cognitive  engagement  in  the  areas  of 
curiosity,  focus,  and  interest;  and  affective  and  cognitive  attitudes.  The  most  significant 
findings  were  the  strong  relationships  present  between  collaborative  adaptability  and  the 
cognitive  processes.  Collaborative  adaptability  was  related  to  curiosity  (p  <  0.01,  r  = 
0.785),  focus  (p  <  0.01,  r  =  0.766),  interest  (p  <  0.05,  r  =  0.607),  affective  attitude  (p  < 
0.01,  r  =  0.759),  and  cognitive  attitude  (p  <  0.01,  r  =  0.727). 

The  data  suggests  that  the  most  influential  competency  that  affects  a  student’s 
cognitive  processes  is  collaborative  adaptability.  Relationships  between  cognitive 
processes  and  both  autonomous  learning  and  multiple  level  operation  were  not  found  to 
be  significant.  Nevertheless,  due  to  the  inherent  difficulty  of  obtaining  significant 
relationships  with  such  a  small  sample  size,  we  conducted  further  descriptive  analysis 
regarding  multiple  level  operations  and  autonomous  learning.  We  divided  the  sample 
according  to  highest  and  lowest  quartiles  in  terms  of  the  students’  rankings  of  both 
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autonomous  learning  and  multiple  level  operations.  We  then  found  the  average  score  for 
these  groups  for  each  of  the  five  competencies  of  cognitive  process  to  determine  the 
nature  of  the  relationship  between  autonomous  learning  and  cognitive  process  and 
multiple  level  operations  and  cognitive  processes.  We  found  that  students  in  the  top 
quartile  with  respect  to  multiple  level  operation  later  rated  their  cognitive  engagement 
and  attitudes  more  highly  compared  to  the  last  quartile  of  students.  This  suggests  that 
higher  multiple  level  operation  is  likely  to  enhance  the  cognitive  processes.  Figure  6 
illustrates  the  top  quartile  (Ql)  and  the  bottom  quartile  (Q4)  students  with  respect  to 
multiple  level  operation  and  their  corresponding  affective  and  cognitive  attitude  scores. 
Figure  7  shows  the  relationships  between  multiple  level  operation  and  the  three  forms  of 
cognitive  engagement.  Similarly,  Figures  8  and  9  reflect  the  top  and  bottom  quartiles 
with  respect  to  autonomous  learning  and  their  associated  attitude  and  cognitive 
engagement  scores  as  well. 
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Figure  6.  Multiple  Level  Operation  and  Cognitive  Attitudes 
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Cognititve  Engagement 
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Autonomous  Learning  and  Cognitive  Engagement 
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Figure  9.  Autonomous  Learning  and  Cognitive  Engagement 

2.  Learning  Outcomes 

Our  analysis  of  the  correlation  data  also  determined  relationships  that  influence  a 
student’s  achievement  of  learning  outcomes.  Final  performance  score  and  course 
satisfaction  represent  important  learning  outcomes  set  forth  by  the  course  objectives. 
Strong  relationships  were  exhibited  between  collaborative  adaptability,  again,  and 
learning  outcomes.  Collaborative  adaptability  was  positively  related  to  final  score  (p  < 
0.05,  r  =  0.423)  and  satisfaction  (p  <  0.01,  r  =  0.747).  All  dimensions  of  cognitive 
engagement  (curiosity:  p  <  0.01,  r  =  0.71;  attention:  p  <  0.05  r  =  0.51;  interest:  p  <  0.01,  r 
=  0.82)  and  attitude  (affective:  p  <  0.01,  r  =  0.83;  cognitive:  p  <  0.01,  r  =  0.76)  were  also 
positively  related  to  student  satisfaction.  Interesting  though,  none  of  the  cognitive  process 
was  significantly  associated  with  the  student’s  final  score.  Hence,  in  the  VSH,  the 
cognitive  processes  are  very  important  to  enhancing  student  satisfaction  with  the  course, 
but  appear  to  have  less  impact  on  their  test  scores. 


29 


The  data  suggests  collaborative  adaptability  also  strongly  influences  a  student’s 
ability  to  successfully  pass  the  final  performance  examination  and  their  satisfaction  of  the 
course.  This  leads  us  to  determine  that  collaborative  adaptability  is  the  primary  driver  of 
success  in  the  VSH,  directly  relating  to  student’s  cognitive  processes  and  learning 
outcomes.  Relationships  between  learning  outcomes  and  both  autonomous  learning  and 
multiple  level  operation  were  not  found  to  be  significant.  Again,  due  to  the  inherent 
difficulty  of  obtaining  significant  relationships  with  such  a  small  sample  size,  we 
conducted  further  descriptive  analysis  regarding  multiple  level  operations  and 
autonomous  learning. 

As  with  their  relationships  with  the  cognitive  processes,  and  as  reflected  in 
Figures  10  and  11,  the  data  suggests  that  these  relationships  also  trend  in  the  expected 
directions  as  illustrated  in  our  model. 
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Figure  10.  Competencies  and  Satisfaction 
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V.  DISCUSSION  AND  CONCLUSION 


A.  SUMMARY 

While  Navy  budget  constraints  are  increasing  with  continuing  resolve,  the 
demand  for  more  efficient  and  effective  training  is  also  increasing  as  the  Navy  strives  for 
professional,  proficient  sailors.  Previous  research  conducted  at  Naval  Postgraduate 
School  (NPS)  has  resulted  in  a  wealth  of  understanding  regarding  the  unique  student 
competencies  that  may  help  enhance  learning  using  virtual  classroom  technology. 

Our  goal  for  this  study  was  to  extend  previous  Spears  and  Aten-DiRenzo  research 
findings  to  further  clarify  and  enhance  the  VSH  learning  competencies  model  (Aten  & 
DiRenzo,  2014;  Spears,  2014).  The  specific  goal  of  our  research  was  to  refine,  introduce, 
and/or  identify  student  success  factors  and  competencies  in  a  synchronous  distributed 
virtual  learning  environment. 

In  order  to  collect  the  relevant  student  data,  we  conducted  three  simultaneous 
onsite  visits  to  VSH  locations  in  Norfolk,  Virginia;  Groton,  Connecticut;  and  Kings  Bay, 
Georgia.  While  at  these  locations,  we  used  a  refined  list  of  survey  questions  derived  from 
our  literature  review  to  administer  anonymous  pre-  and  post-surveys,  in  addition  to  obtain 
preliminary  agreements  for  follow-on  oral  interviews  in  order  to  collect  data  in  three 
areas.  First,  we  wanted  to  identify  which  features  and  tools  students  found  either 
beneficial  or  detrimental  to  their  VSH  experience.  Second,  we  wanted  to  examine  how 
instructor  features  and  tools  contribute  to  a  student’s  VSH  learning  experience.  Third,  we 
wanted  to  identify  new,  as  well  as  verify  existing,  student  learning  competencies  that 
could  contribute  to  a  student’s  success  in  VSH  courses.  Our  overall  intent  was  to  refine 
and  simplify,  if  possible,  the  Aten-DiRenzo  model  for  learning  in  the  virtual  world  to 
reflect  our  findings. 

A  quantitative  analysis  was  conducted  based  on  the  data  gathered  from  pre-  and 
post-class  surveys  that  were  originally  developed  by  Aten  and  DiRenzo,  which  we 
expanded  with  additional  questions  inspired  by  our  literature  review  (2014).  Our 
statistical  and  literature  review  of  the  seven  competencies  tested — adaptability, 
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ambiguity,  articulation,  interactional,  multiple  levels,  operational,  and  self-regulated 
learning — led  to  our  findings  in  terms  of  new  or  refined  competencies  and  subsequently 
the  Bower- Richmond  Model  for  student  learning  competencies  in  the  virtual 
environment. 

Our  first  finding  was  that  a  student’s  virtual  world  efficacy — one’s  capability  to 
understand,  operate,  and  learn  in  the  virtual  world — strongly  correlated  with  nearly  every 
other  attribute,  indicating  that  this  broad  competency  was  the  single  most  significant 
predictor  of  a  student’s  success  in  achieving  desired  learning  outcomes  in  the  VSH.  It 
was  determined  that  virtual  world  efficacy  would  be  our  top-tier  competency  from  which 
all  other  competencies  would  be  derived.  Virtual  world  efficacy  also  positively 
influenced  the  student’s  applications  of  the  operational,  interactional,  and  self-regulated 
learning  competencies. 

Stemming  from  virtual  world  efficacy  are  our  second-tier  learning  competencies 
and  their  associated  subset  competencies.  This  is  illustrated  in  Figure  4  by  the  solid  lines 
from  “Virtual  World  Efficacy”  to  the  three  identified  competencies,  re-conceptualized 
them  to  “Autonomous  Learning,”  “Multiple  Level  Operation,”  and  “Collaborative 
Adaptability.”  Lirst,  the  operational  competency  statistically  correlated  with  multiple 
levels  and  ambiguity  competencies.  Incorporating  the  definitions  of  multiple  levels  and 
ambiguity,  we  revised  the  operational  competency  as  “multiple  level  operation.”  We 
expanded  the  operational  competency  to  encompass  a  student’s  ability  to  operate  and 
communicate  in  multiple  levels  in  the  VSH,  while  accepting  contextual  ambiguity  in  the 
virtual  world. 

Next,  the  interactional  competency  statistically  correlated  with  articulation, 
adaptability,  and  synchronous  chat  competencies.  We  merged  the  definitions  of 
articulation  in  the  virtual  environment,  adaptability,  and  synchronous  chat  to  revise  the 
interactional  competency.  We  then  chose  to  revise  the  interactional  competency 
“collaborative  adaptability”  to  accurately  describe  the  tenets  of  the  competency. 

Linally,  self-regulated  learning  correlated  logically  with  the  overall  concept  of 
learning  in  the  virtual  environment.  Students  must  possess  an  appropriate  level  of 
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autonomy  in  order  to  learn  independently,  without  prompting  from  instructors,  and 
proactively  so  they  can  achieve  the  desired  learning  outcomes.  We  revised  self-regulated 
learning  “autonomous  learning,”  which  encompasses  a  student’s  self-directed  learning 
approaches. 

Collectively,  these  correlations  led  to  the  development  of  the  Bower-Richmond 
model  for  student  learning  competencies  in  the  virtual  learning  environment.  The  model 
illustrates  virtual  world  efficacy  as  the  meta-competency,  followed  by  the  secondary 
competencies  of  autonomous  learning,  multiple  level  operation,  and  collaborative 
adaptability.  These  learning  competencies,  together,  stimulate  a  student’s  cognitive 
processes  of  cognitive  engagement  and  attitude,  as  shown  in  Figure  4  by  the  bold  bracket 
pointing  from  the  three  learning  competencies  towards  the  “Cognitive  Processes”  box.  In 
Figure  4,  a  solid  arrow  leads  from  the  cognitive  processes  to  the  “Learning  Outcomes” 
box,  illustrating  a  direct  influence.  Cumulatively,  the  learning  competencies  and 
cognitive  processes  aid  in  a  student’s  ability  to  successfully  achieve  the  desired  learning 
outcomes,  in  terms  of  measured  performance  and  student  satisfaction,  for  a  virtual 
learning  course.  Lastly,  from  the  “Satisfaction”  subset  in  the  “Learning  Outcomes”  box  in 
Figure  4,  a  dashed- line  arrow  shows  a  reflective  relationship  to  a  student’s  attitude 
towards  the  virtual  learning  course. 

B.  CONCLUSION 

The  realized  cost  savings  and  proven  effectiveness  of  VSH  learning  will,  in  all 
likelihood,  remain  part  of  the  Navy’s  learning  structure  and  likely  grow  in  scope  as 
technology  matures  and  entrenched  military  paradigms  fade  or  shift.  While  certain  doubt 
remains  among  the  more  traditional  elements  within  the  military,  the  benefits  of  virtual 
learning  are  clear.  However,  some  doubt  remains  regarding  the  success  of  VSH  training 
in  terms  of  student  success,  whether  that  success  is  measured  as  the  students’  satisfaction 
with  the  course  or  in  terms  of  test  scores.  Our  research  suggests  that  students  can  be 
successful  in  the  VSH  both  in  terms  of  satisfaction  and  performance,  and  there  are  certain 
steps  that  can  be  taken  by  the  SLC  and  the  Navy  to  assist  students  in  achieving  both 
outcomes. 
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Our  research  and  findings  indicate  that  a  student’s  capability  to  understand  and 
learn  in  the  virtual  world  is  a  significant  predictor  of  his  or  her  success  in  the  virtual 
environment.  We  have  defined  this  broad  competency  as  “virtual  world  efficacy,”  a 
student’s  ability  to  leam  and  operate  in  the  virtual  world,  and  believe  it  is  a  fundamental, 
top-tier  input  to  the  Bower- Richmond  model  for  student  learning  in  the  virtual 
environment  and  is  illustrated  in  Figure  4.  Given  the  importance  of  this  competency,  we 
want  to  illustrate  which  steps  can  be  taken  to  assist  the  student  in  developing  his  or  her 
virtual  world  efficacy  with  the  intent  of  increasing  their  satisfaction  with  the  class  and 
their  performance  in  general. 

A  preliminary  review  of  responses  to  several  of  the  short  answer  questions  on  the 
post-class  surveys  and  the  oral  interviews  suggests  that  most  students  indicated  that  a 
block  of  time  dedicated  to  learning  the  basic  functionally  of  operating  and  interacting  in 
the  VSH  would  have  contributed  to  their  achieving  greater  success  in  the  class.  Providing 
a  “familiarization  period”  either  at  the  beginning  of  the  first  class  or  a  day  or  two  prior  to 
the  VSH  course  convening  would  enhance  a  student’s  virtual  world  efficacy  and  lead  to 
greater  success  in  the  classroom.  When  assessing  some  of  the  second-tier  competencies 
or  outputs  of  virtual  world  efficacy,  it  is  clear,  that  a  block  of  time  dedicated  to 
familiarizing  students  with  synchronous  chat,  articulation,  and  general  technical  abilities 
terms  of  competencies  can  affect  other  competencies  such  as  self-directed  learning, 
ambiguity  (or  the  removal  of  ambiguity),  proactivity,  and  confidence.  The  more 
comfortable  a  student  feels  prior  the  class  commencing,  when  there  is  pressure  to 
participate  and  perform,  the  more  likely  he  or  she  is  able  to  achieve  a  level  of  virtual 
world  efficacy  that  will  allow  him  or  her  to  enjoy  and  perform  well  in  the  course.  Even 
the  competency,  such  as  synchronous  chat,  has  the  potential  to  contribute  enough  to  a 
student’s  virtual  world  efficacy  to  influence  his/her  success. 

The  effectiveness  of  a  familiarization  period  prior  to  commencing  the  actual 
course  is  dependent  on  some  very  basic  functions  that  are  unique  to  all  VSHs  in  addition 
to  some  that  are  unique  to  the  SLC  and  the  military  in  general.  Based  on  our  personal 
observations  in  the  three  separate  classrooms,  there  were  significant  technical  difficulties 
that  hindered  and  sometimes  stopped  students’  entrance  into  the  VSH.  These  external 

36 


factors,  while  not  related  to  the  course  content,  course  material,  or  even  the  virtual  world, 
had  significant  impacts  on  students’  ability  to  gain  access  to  the  virtual  world,  familiarize 
themselves,  get  comfortable,  and  ultimately  increase  their  virtual  world  efficacy. 

The  most  common  issues  were  both  technical  and  administrative.  For  example, 
some  students  did  not  have  the  proper  log-in  to  gain  access  to  the  computer  needed  for 
the  course.  In  other  situations,  some  students  encountered  issues  with  security  clearance 
verification  and  other  “paperwork”  processes  that  delayed  their  access. 

While  not  as  common,  there  were  also  instances  where  hardware,  such  as 
headphones,  keyboards,  and  monitors,  would  fail.  These  hardware  failures  caused 
additional  stress,  which  had  the  greatest  impact  on  students  already  deficient  in  their 
virtual  world  efficacy.  In  order  to  achieve  the  most  virtual  world  efficacy  from  each 
student,  the  SLC  should  make  every  attempt  to  ensure  that  all  administrative,  technical, 
and  hardware  issues  are  mitigated  to  the  maximum  extent  possible  prior  to  the  beginning 
of  the  class.  Avoiding  instances  that  the  instructor  has  to  pause  to  clear  up  a  student’s 
confusion  or  frustration  with  some  aspect  of  the  course  that  is  not  content  related  or  VSH 
related  gives  all  students,  and  in  particular  students  with  a  lower  virtual  world  efficacy, 
the  best  chance  of  achieving  success  in  the  VSH. 

C.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

The  study  of  synchronous  dispersed  learning  in  in  a  military  setting  is  still  in  its 
relative  infancy.  The  dramatic  increase  in  popularity  of  asynchronous  dispersed  learning 
institutions,  such  as  the  University  of  Phoenix  and  a  growing  number  of  public 
institutions,  has  generated  a  plethora  of  available  research  that  has  provided  a  valuable 
baseline  for  the  SLC,  Navy,  and  military  in  general  to  begin  to  understand  its  applications 
in  military  training.  However,  there  still  remains  a  dearth  of  research  on  synchronous 
learning  in  the  virtual  world.  This  is  especially  true  for  military  applications.  Most  studies 
still  focus  on  synchronous  gaming  in  the  virtual  world  and  not  an  academic  setting.  Given 
this  lack  of  research  and  other  differences  inherent  to  the  Navy  and  military,  we 
recommend  that  the  Navy  continue  to  provide  access  to  study,  and  continue  to  refine  and 
enhance  the  VSH  competency  model. 
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First,  based  on  the  data  and  our  observations  we  determined  that  virtual  world 
efficacy  was  a  broad  overarching  competency  from  which  comprised  other  competencies 
such  as  articulation,  synchronous  chat,  ability,  etc.  We  also  speculated  that  a 
“familiarization  period”  before  commencement  of  class  has  the  potential  to  affect  some 
of  the  second  tier  competencies,  essentially  affecting  the  top  tier  competency  of  virtual 
world  efficacy,  leading  to  greater  student  and  satisfaction  and  overall  performance.  The 
SLC  and  the  Navy  could  gain  significant  insight  by  testing  in  greater  size  and  scale  the 
effects  of  a  familiarization  period  on  student’s  second  tier  competencies  according  to  the 
Bower  Richmond  model  and  measuring  if  the  development  of  one  or  several  of  these 
competencies  had  a  positive  effect  on  the  student’s  virtual  world  efficacy  and  his  or  her 
satisfaction  and  performance  in  the  course. 

Second,  additional  steps  should  also  investigate  interactivity  and  collaboration  in 
terms  student-to- student  competencies  as  well  as  the  student-to-system  and  student-to- 
instructor  competencies,  previously  included  in  the  Spears  model,  as  potential  predictors 
of  student  success  (2014).  Student-to-student  interactivity  and  collaboration  are 
increasingly  vital  forms  of  learning  and  problem  solving  in  the  classroom  and  workplace. 
Future  studies  could  include  enhanced  opportunities  for  students  to  work  together  on 
specific  problem-solving  or  decision-making  situations  related  to  the  class  material.  Then 
students’  reaction  to  these  student-to-student  learning  opportunities  could  be  measured 
and  evaluated.  These  student-to-student  research  results  could  then  be  used  to  refine  the 
current  Bower  Richmond  VSH  Competencies  Model. 

Next,  while  not  a  student  competency,  instructor  competencies  are  closely  related 
and  should  also  be  measured  in  future  research.  Aten  and  DiRenzo’s  research  included 
several  indications  that  instructor  competencies  can  significantly  impact  each  student’s 
learning  experience  in  the  VSH  and  the  authors  suggested  priming  designed  to  enhance 
students’  perceptions  of  instructor  competency  be  included  in  future  research  (2014). 
Instructor  competency  is  a  vital  key  to  the  success  of  any  learning  experience  and  this 
competency  may  be  even  more  important  in  remote  learning  settings  like  VSH. 

Additionally,  our  findings  did  not  identify  significant  relationships  that  show 

influence  between  autonomous  learning  and  multiple  level  operation  and  cognitive 
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processes  and/or  learning  outcomes.  We  predict  this  was  due  to  our  small  sample  size. 
However,  this  could  be  due  the  nature  of  a  military  environment,  where  rank  structure 
and  rigid  adherence  to  classroom  procedures  may  have  affected  these  competencies. 
Likewise,  since  the  VSH  is  a  synchronous  environment,  perhaps  autonomous  learning 
and  multiple  level  operations  are  not  as  profound  and  significant  competencies  that 
contribute  to  a  student’s  cognitive  processes  and  overall  achievement  of  learning 
outcomes.  Thus,  we  suggest  expansion  from  this  study  to  obtain  a  greater  sample  size  to 
further  identify  and  explore  such  relationships. 

Follow-on  research  with  broad  access  to  students  and  consistent  monitoring  of 
student  success  in  the  virtual  classroom  is  not  only  critical  to  a  successful  classroom 
learning  experience,  but  is  also  vital  to  the  safe  and  effective  operation  of  submarines 
and  all  other  U.S.  naval  assets.  It  is  essential  that  the  while  offering  a  cost  savings,  the 
SLC  and  the  Navy  ensure  service  members  have  the  best  training  and  chance  to  succeed 
in  the  VSH. 
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APPENDIX  A.  PRE-TRAINING  SURVEY  QUESTIONNAIRE 


SECTION  1 

Please  circle  your  response  to  the  following  questions. 

How  many  times  in  the  past  year  have  you  done  the  following: 

1.  Played  a  PC  based  video  game? 

Never  Seldom  Sometimes  Frequently  Often 

2.  Played  a  console  video  game  system  (e.g.,  XBox,  PlayStation  3,  Wii,  etc...)? 

Never  Seldom  Sometimes  Frequently  Often 

3.  During  an  average  week,  how  many  hours  will  you  spend  playing  video  games  ? 

<  1  hour  1-3  hours  3-5  hours  5-7  hours  7-9  hours  >  9hours 

4.  I  consider  myself: 

A.  A  non-video  game  player 

B.  A  novice  video  game  player 

C.  An  occasional  video  game  player 

D.  A  frequent  video  game  player 

E.  An  expert  video  game  player 

SECTION  2 

Please  respond  by  circling  the  number  that  corresponds  with  the  extent  to  which 
you  agree  with  the  following  statements  on  a  scale  from  1  =  strongly  disagree  to  5  = 
strongly  agree. 

Many  of  the  following  statements  are  very  similar  to  one  another.  This  is  necessary 
to  ensure  the  validity  of  the  study’s  results. 


1. 1  am  confident  I  get  the  success  I  deserve  in  life. 

Strongly  Disagree  1  2  3  4  5 

Strongly  Agree 

2.  When  I  try,  I  generally  succeed. 

Strongly  Disagree  12  3 

4  5 

Strongly  Agree 
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3. 1  complete  tasks  successfully. 

Strongly  Disagree  1  2  3  4  5 

4.  Overall,  I  am  satisfied  with  myself. 

Strongly  Disagree  1  2  3  4  5 

5. 1  determine  what  will  happen  in  my  life. 

Strongly  Disagree  1  2  3  4  5 

6. 1  am  capable  of  coping  with  most  of  my  problems. 

Strongly  Disagree  1  2  3  4  5 

SECTION  3 


Strongly  Agree 


Strongly  Agree 


Strongly  Agree 


Strongly  Agree 


1.  I  prefer  to  do  things  that  I  can  do  well  rather  than  things  that  I  do  poorly. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  I’m  happiest  at  work  when  I  perform  tasks  on  which  I  know  that  I  won’t  make  any 
errors. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  The  opportunity  to  do  challenging  work  is  important  to  me. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  When  I  fail  to  complete  a  difficult  task,  I  plan  to  try  harder  the  next  time  I  work  on  it. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

5.  I  prefer  to  work  on  tasks  that  force  me  to  leam  new  things. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

6.  The  things  I  enjoy  the  most  are  the  things  I  do  the  best. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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7.  The  opinions  others  have  about  how  well  I  can  do  certain  things  are  important  to  me. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

8.  I  feel  smart  when  I  do  something  without  making  any  mistakes. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

9.  I  like  to  be  fairly  confident  that  I  can  successfully  perform  a  task  before  I  attempt  it. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

10.  The  opportunity  to  learn  new  things  is  important  to  me. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

1 1. 1  do  my  best  when  I’m  working  on  a  fairly  difficult  task. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

12. 1  try  hard  to  improve  on  my  past  performance. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

13.1  like  to  work  on  tasks  that  I  have  done  well  on  in  the  past. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

14. 1  feel  smart  when  I  can  do  something  better  than  most  other  people. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

15.  The  opportunity  to  extend  the  range  of  my  abilities  is  important  to  me. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

16.  When  I  have  difficulty  solving  a  problem,  I  enjoy  trying  different  approaches  to  see 
which  one  will  work. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  4 

1.  Compared  to  most  people,  I  am  typically  unable  to  get  what  I  want  out  of  life? 
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Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  Growing  up,  I  would  “cross  the  line”  by  doing  things  that  my  parents  would  not 
tolerate? 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3. 1  have  often  accomplished  things  that  got  me  "psyched"  to  work  even  harder? 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4. 1  got  on  my  parents’  nerves  often  when  I  was  growing  up? 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

5. 1  obeyed  rules  and  regulations  that  were  established  by  your  parents? 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

6.  Growing  up,  I  often  acted  in  ways  that  my  parents  thought  were  objectionable? 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

7.  Do  you  often  do  well  at  different  things  that  you  try? 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

8.  Not  being  careful  enough  has  gotten  me  into  trouble  at  times. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

9.  When  it  comes  to  achieving  things  that  are  important  to  me,  I  find  that  I  don't  perform 
as  well  as  I  ideally  would  like  to  do. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

10. 1  feel  like  I  have  made  progress  toward  being  successful  in  my  life. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

1 1. 1  have  found  very  few  hobbies  or  activities  in  my  life  that  capture  my  interest  or 
motivate  me  to  put  effort  into  them. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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SECTION  5 


1.  During  class  time  I  often  miss  important  points  because  I’m  thinking  of  other  things. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  When  reading  for  this  course,  I  make  up  questions  to  help  focus  my  reading. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  When  I  become  confused  about  something  I’m  reading  for  this  class,  I  go  back  and  try 
to  figure  it  out. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  If  course  materials  are  difficult  to  understand,  I  change  the  way  I  read  the  material. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

5.  Before  I  study  new  course  material  thoroughly,  I  often  skim  it  to  see  how  it  is 
organized. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

6. 1  ask  myself  questions  to  make  sure  I  understand  the  material  I  have  been  studying  in 
this  class. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

7. 1  try  to  change  the  way  I  study  in  order  to  fit  the  course  requirements  and  instructor’s 
teaching  style. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

8. 1  often  find  that  I  have  been  reading  for  class  but  don’t  know  what  it  was  all  about. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

9. 1  try  to  think  through  a  topic  and  decide  what  I  am  supposed  to  learn  from  it  rather  than 
just  reading  it  over  when  studying. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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10.  When  studying  for  this  course  I  try  to  determine  which  concepts  I  don’t  understand 
well. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

11.  When  I  study  for  this  class,  I  set  goals  for  myself  in  order  to  direct  my  activities  in 
each  study  period. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

12.  If  I  get  confused  taking  notes  in  class,  I  make  sure  I  sort  it  out  afterwards. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  6 
I  would  feel  confident... 

1.  Opening  a  web  browser  (e.g.  Firefox  or  Explorer). 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  Reading  text  from  a  website. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  Clicking  on  a  link  to  visit  a  specific  website. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  Conducting  an  Internet  search  using  one  or  more  keywords. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

5.  Creating  a  simple  web  page  with  text,  images,  and  links. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  7 
I  would  feel  confident... 

1.  Providing  a  nickname  within  a  synchronous  chat  system  (if  necessary). 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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2.  Reading  messages  from  one  or  more  members  of  the  synchronous  chat  system. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  Answering  a  message  or  providing  my  own  message  in  a  synchronous  chat  system 
(one-to-many  interaction). 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  Interacting  privately  with  one  member  of  the  synchronous  chat  system  (one-to-one 
interaction). 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  8 
I  would  feel  confident... 

1.  Moving  my  avatar  in  a  virtual  world  or  a  computer  simulation. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  Adjusting  my  camera  angle  in  a  virtual  world  or  a  computer  simulation. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  Adjusting  the  view  of  my  avatar  in  a  virtual  world  or  a  computer  simulation. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  Focusing  on/zooming  in  on  an  object  in  a  virtual  world  or  a  computer  simulation. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

5.  Navigating  around  objects  in  a  virtual  world  or  a  computer  simulation. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  9-A 

1. 1  am  comfortable  using  different  technological  tools  in  a  virtual  classroom. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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2. 1  can  excel  in  a  virtual  classroom  because  I  know  how  to  use  many  different 
technological  tools. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3. 1  do  NOT  know  how  to  use  virtual  classroom  tools. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  9-B 

1. 1  am  confident  I  can  use  technology  (email,  chat,  virtual  environments,  etc.)  to 
communicate  effectively  with  others. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2. 1  am  confident  I  can  use  technology  (email,  chat,  virtual  environments,  etc.)  to 
interactively  solve  problems  with  others. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3. 1  can  get  work  done  with  others  using  technology  (email,  chat,  virtual  environments, 
etc.)  just  as  well  as  I  can  when  working  face-to-face. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  10-A 

1. 1  consciously  articulate  my  thoughts  differently  online  as  opposed  to  in  person. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2. 1  adjust  my  approach  to  communicate  with  others  online. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3. 1  am  more  deliberate  in  the  ways  I  communicate  and  express  my  thoughts  online. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  10-B 

1. 1  find  it  easy  to  switch  between  multiple  conversations  in  a  virtual  world. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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2. 1  can  handle  multiple  tasks  simultaneously  in  a  virtual  world. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3. 1  feel  comfortable  conversing  over  multiple  channels  (e.g.,  chat,  voice,  email,  text,  etc.) 
at  the  same  time. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  10-C 

1. 1  feel  comfortable  learning  in  a  digital  world. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2. 1  feel  comfortable  seeking  additional  clarification  when  a  situation  in  the  virtual  world 
confuses  me. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3. 1  feel  comfortable  dealing  with  the  ambiguity  associated  with  learning  in  a  virtual 
world. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  10-D 

1. 1  enjoy  changes  in  job  policies,  procedures  or  practices. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2. 1  like  it  when  the  structure  of  my  work  changes. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  It  is  easy  for  me  to  adapt  when  my  job  tasks  change. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4. 1  appreciate  when  the  work  I  do  in  my  job  changes. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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APPENDIX  B.  POST-TRAINING  SURVEY  QUESTIONNAIRE 


Please  respond  by  circling  the  number  that  corresponds  with  the  extent  to  which 
you  agree  with  the  following  statements  on  a  scale  from  1  =  strongly  disagree  to 
5  =  strongly  agree. 

Many  of  the  following  statements  are  very  similar  to  one  another.  This  is  necessary 
to  ensure  the  validity  of  the  study’s  results. 

SECTION  1 

1. 1  had  numerous  interactions  with  the  instructor  during  the  class. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2. 1  asked  the  instructor  my  questions  through  different  electronic  means,  such  as  email, 
discussion  board,  instant  messaging  tools,  etc. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  The  instructor  regularly  posted  some  questions  for  students  to  discuss  on  the  discussion 
board. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  The  instructor  replied  my  questions  in  a  timely  fashion. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

5. 1  replied  to  messages  from  the  instructor. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

6. 1  received  enough  feedback  from  my  instructor  when  I  needed  it. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  2 

1.  Virtual  schoolhouse  course  materials  helped  me  to  understand  the  class  content. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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2.  Virtual  schoolhouse  course  materials  stimulated  my  interest  in  this  course. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  Virtual  schoolhouse  course  materials  helped  me  relate  my  personal  experience  to  new 
concepts  or  new  knowledge. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  It  was  easy  for  me  to  access  the  virtual  schoolhouse  course  materials. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  3-A 

1.  The  Virtual  School  House  excites  my  curiosity. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  The  Virtual  School  House  arouses  my  imagination. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  Learning  with  the  Virtual  School  House  makes  me  curious. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  3-B 

1.  When  using  the  Virtual  School  House,  I  am  totally  absorbed  in  what  I  am  learning. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  The  Virtual  School  House  holds  my  attention. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  When  learning  with  the  Virtual  School  House,  I  am  aware  of  distractions. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  3-C 

1.  The  Virtual  School  House  is  fun. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 


52 


2.  The  Virtual  School  House  is  interesting. 


Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  The  Virtual  School  House  is  boring. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  4-A 

1.  Using  the  Virtual  School  House  is  pleasurable. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  Using  the  Virtual  School  House  is  a  positive  experience. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3. 1  enjoy  using  the  Virtual  School  House. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  4-B 

1.  The  Virtual  School  House  is  a  useful  tool  for  learning  tasks  related  to  my  job. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  The  Virtual  School  House  is  beneficial. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  The  Virtual  School  House  is  a  valuable  resource  for  learning  new  skills. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  5 

1.  The  course  instructors  are  competent  teachers. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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2. 1  learned  a  lot  because  of  the  quality  of  the  course  instructors. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3. 1  would  recommend  these  course  instructors  to  future  students. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  6 

1.  The  course  instructors  were  passionate  about  this  course. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  The  course  instructors  seemed  to  enjoy  teaching  this  course. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  The  course  instructors  had  a  positive  view  of  this  course. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  The  instructors  valued  this  course. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  7 

1.  Using  the  Virtual  School  House  increased  my  learning  and  productivity. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  Using  the  Virtual  School  House  improved  my  performance  and  skills. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  Using  the  Virtual  School  House  enhanced  my  effectiveness. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4. 1  find  the  Virtual  School  House  useful  to  my  learning  and  development. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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SECTION  8 


1.  Learning  to  operate  the  Virtual  School  House  is  easy  for  me. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2. 1  find  it  easy  to  get  the  Virtual  School  House  to  do  what  I  want  it  to  do. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  It  would  be  easy  for  me  to  become  skillful  at  using  the  Virtual  School  House. 
Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4. 1  find  the  Virtual  School  House  easy  to  use. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  9 

1.  The  course  instructors  were  passionate  about  the  Virtual  School  House. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

2.  The  course  instructors  seemed  to  enjoy  using  the  Virtual  School  House. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  The  course  instructors  had  a  positive  view  of  the  Virtual  School  House. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4.  The  instructors  valued  the  Virtual  School  House. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

SECTION  10 

1.  Overall,  I  am  satisfied  with  this  course. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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2.  This  course  contributed  to  my  educational  development. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

3.  This  course  contributed  to  my  professional  development. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

4. 1  am  satisfied  with  the  level  of  interaction  that  happened  in  this  course. 
Strongly  Disagree  1  2  3  4  5  Strongly  Agree 

5.  In  the  future,  I  would  be  willing  to  take  a  course  like  this  again. 

Strongly  Disagree  1  2  3  4  5  Strongly  Agree 
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APPENDIX  C.  INTERVIEW  QUESTIONS 


Describe  your  experience  overall. 

What  tools  did  find  the  most  useful? 

When  did  you  find  you  would  have  preferred  face-to-face  training? 

What  features  or  tools  were  distracting? 

What  features  would  you  have  liked  to  have  had? 

Thinking  back  on  the  training,  can  you  tell  me  about  a  time  that  you  were  confused? 
What  happened?  How  did  you  overcome  it? 

What  contributed  most  to  your  learning  in  the  VSH? 

How  was  this  different  from  face-to-face  training? 

What  was  distracting  or  difficult  in  the  VSH? 

How  was  this  different  from  the  face-to-face  training? 

Can  you  give  me  an  example  of  a  time  that  something  didn’t  work  for  you  in  the  VSH? 
An  example  of  something  that  didn’t  work  in  the  previous  face-to-face? 

Can  you  think  about  something  that  worked  really  well  in  the  VSH?  What  happened? 
What  features  did  you  use? 

What  was  missing  (compared  to  face-to-face)? 

What  was  added  or  better  (compared  to  face-to-face)? 

Which  features  were  most  useful? 

Which  features  were  least  useful? 

Which  were  distracting? 

What  did  you  most  enjoy? 

What  did  you  least  enjoy? 

What  would  you  change? 

How  could  it  be  improved? 

How  will  this  experience  influence  your  interest/ability  in  using  the  VSH  in  the  future? 
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APPENDIX  D.  CORRELATION  DATA 


Table  2 


Correlation  Data  between  All  Learning  Competencies 
Derived  from  Student  Surveys  (where  p- value  <  0.05) 
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